Mapsforge (Coastline-Enabled) Wiki
Introduction This wiki provides documentation of the coastline-aware mapfile-writer. Correct handling of coastlines has long been missing in mapsforge. We now extended the original mapfile-writer to solve this issue. The extended version allows for two different modes of processing coastline-data. The native mode tries to create water-polygons from the data in the input file alone, the shapefile mode completely neglects coastline information from the input file, but uses external water-polygons provided as a shapefile. Conceptual Overview A few versions ago the mapsforge library lost its ability to process and display water areas defined by coastlines. Since then coastlines were only present in form of blue lines without the actual water areas. The reason for this was twofold: First, the processing of coastlines was not sufficiently robust, many errors displayed in the rendered maps, with some land areas being rendered as water areas and vice versa. Second, processing of coastlines was part of the rendering engine. Complicated computational geometry routines were run on the mobile device. We decided to completely rewrite the code so that all coastline processing takes place "offline" as part of map file generation. We now compute water polygons in advance and simply store them as regular (closed) ways within the map file. The new coastline-aware mapfile-writer computes these polygons by performing an additional preprocessing phase. Since the preprocessed coastline information is saved within the regular mapfile, no additional adjustments or processing steps are needed in a client application, resulting in a much smoother performance. In OpenStreetMap coastlines are handled differently than other entities. For example, the coastlines' direction determines which area is water and which is land (as explained here.) To make matters worse, the coastline data contains many errors violating some properties of a general contract for coastlines. In consequence, extra care is necessary to get things right. As mentioned earlier, we implemented two major modes to generate the necessary coastline geometries. The native mode utilizes the coastline data that is included in the OSM input file. The shapefile mode takes as an additional input an already preprocessed shapefile with coastline geometries. This is typically provided by an external source such as openstreetmapdata.com. In general the shapefile mode is assumed to be more reliable (data already corrected) and efficient (in terms of performance), whereas the native mode does not rely on external data and thus may be more consistent with the input file (shapefiles might be stale). A major drawback of the native mode is, that it requires complete coastline data over the rectangular bounding-box implicitly defined by the input file. For example, if you try to convert a file which was clipped using a bounding polygon instead of a rectangular bounding box (as most files from Geofabrik), you are guaranteed to fail, because the necessary coastlines have been clipped at the administrative borders. Since the PBF-Format only remembers the corresponding bounding-box and not the polygon, the algorithm will fail to evalute a full water-polygon for that extract. Therefore, to use the native mode you either have to clip the desired extract yourself, using a bigger one (or a full planet-dump), or you have to fallback to shapefile mode, which does not rely on the internal coastline data. Hardware and Software Requirements * Preferably 2GB+ RAM * Java 6 * Git and Maven 2 As of now, only the RAM-mode of the Mapsforge-Map-Writer is supported and fully tested! For performance reasons using Java version 7 instead of Java 6 is recommended. If you want to upgrade, admin/root rights are not required (see below). In order to checkout and compile the latest version you will need Git and Maven. Compilation and Setup First of all ensure Git and Maven are set up appropriately and that Java 7 is properly installed. You can test Java, Git and Maven using these commands: > java -version > git -version > mvn -v Check if Maven uses the right Java-Version (it reads JAVA_HOME). Next you will need the latest Osmosis version (greater or equal to 0.41) : NOTE: Using 0.43 with the current Coastline branch (16 Oct, 2013) causes the following exception: : org.java.plugin.PluginLifecycleException: plug-in org.mapsforge.map.mapfile-writer requires plug-in org.openstreetmap.osmosis.core.plugin.Core which is unknown or has incompatible version. : Downgrading to 0.42 solved the issue. To checkout the working copy of the coastline-aware mapfile-writer at Assembla, use this command in a new and empty folder: > git clone git://git.assembla.com/mapsf_coastline.git > cd mapsf_coastline > git checkout -b Coastline origin/Coastline This should download the whole, adapted mapsforge project in a subfolder mapsf_coastline and load and switch to the appropriate branch. Compile mapsforge by navigating into the mapsf_coastline folder if you are not already in it (where the mapsforge-map-writer folder and the root pom.xml '''file are located) and use: > mvn clean install This should build the project, create a file named '''mapsforge-map-writer-0.3.1-SNAPSHOT-jar-with-dependencies.jar and automatically put it in your local osmosis plugins directory where osmosis should find it. Otherwise copy the jar file from the target directory (created by maven) to the appropriate plugin folder. Make sure that any old plugins are deleted. Now you should have everything you need to run the coastline-aware mapfile-writer. Note however, that you may need to update the osmosis script. One way to do this is to add the following lines in the beginning of your osmosis script in Linux: TMP_DIR="/export/local-1/public/coastline/tmp" MAX_MEM="6000M" JVM_OPTIONS="-XX:+AggressiveOpts -XX:+UseCompressedOops -Djava.util.Arrays.useLegacyMergeSort=true" JAVACMD_OPTIONS="$JVM_OPTIONS -Xmx$MAX_MEM -Djava.io.tmpdir=$TMP_DIR" or the batch file in Windows: SET JAVACMD_OPTIONS=-Xmx1300M -XX:+AggressiveOpts -XX:+UseCompressedOops -Djava.io.tmpdir=X:\TEMP -Djava.util.Arrays.useLegacyMergeSort=true Of course you have to change the temp directory to an existing folder on your machine that has enough disk space (in RAM mode however, the temp directory is not used). Also set the maximum memory to something your computer can handle. ~1.5 GB of Java heap space were enough for a coastline geometry that covers whole Greece. In case you wonder, ''-XX:+AggressiveOpts'' enables Java to use (experimental) performance improvements and ''-XX:+UseCompressedOops'' reduces the general object footprint print by utilizing compressed pointers. The last parameter ''-Djava.util.Arrays.useLegacyMergeSort=true'' (new in Java 7) needed to be included for which appears to be a bug in the JTS-Framework if you use Java 7, otherwise it should be ignored by the JVM anyway. Alternatively or as a point of reference you may also want to check some linux scripts that were used for testing. You find them here. They need to be put in the osmosis/bin directory. Usage First you may want to check the original usage instructions for the mapfile-writer here. NOTE: You cannot use osmosis in type=hd mode with the coastline-enabled writer. It will throw a cryptic exception. Use the default (ram mode). The coastline-aware writer currently comes with 3 additionaly parameters: # coastline-preprocessing optional # coastline-mode # shapefile [optional, iff coastline-mode=0] Description as follows: coastline-preprocessing Valid values: true, false Description: enables or disables the new coastline-preprocessing algorithm. Disabling it should completly bypass the new algorithm for testing purposes. Default is on (true). coastline-mode Valid values: 0, 1, 2 Description: Chooses the desired processing mode. If the mode is set to 0 the native algorithm is used. Therefore the shapefile parameter will not be used, even if specified. As explained above this native mode requires '''full coastline data over the rectangular bounding-box implicitly defined by the input file. If using a preclipped third-party input-file, ensure that it was not clipped using a bounding-polygon (like Geofabrik does). A mode set to 1 or 2 activates the shapefile mode. In that case a valid shapefile '''must be provided with the shapefile parameter (read more below). The difference between mode 1 and 2 is that it uses different JTS operations to merge the splitted water polygons into one. Mode 1 uses a buffer-operation that might be quicker but might potentially be more unstable and has a higher memory footprint. Mode 2 uses a cascade-union-operation which exhibits exactly the contrary properties. For now, we recommend using mode 1. shapefile Valid value: ' Shapefiles for water-polygons can be found here . They have been produced with the OSMCoastline tool created by Jochen Topf. You will need the WGS84-projection version. Extract it to a local directory and point the shapefile parameter to the '''.shp file. Examples > . osmosis --rb file=input/malta.osm.pbf --mapfile-writer file=output/malta.map coastline-mode=0 preferred-language=en Converts the file at input/malta.osm.pbf to the file output/malta.map using native-mode (without the shapefile). > . osmosis --rb file=input/monaco.osm.pbf --mapfile-writer file=output/monaco.map coastline-mode=1 shapefile=shapefiles/water_polygons.shp preferred-language=en Converts the file at input/monaco.osm.pbf to the file output/monaco.map using shapefile-mode with the shapefile at shapefiles/water_polygons.shp For clipping a map extract with Osmosis I used something like this: > . osmosis.sh \ > --rb file=input/planet.osm.pbf \ > --buffer bufferCapacity=1000 \ > --bounding-box bottom=43.72 left=7.40 top=43.76 right=7.45 completeWays=yes clipIncompleteEntities=true \ > --buffer bufferCapacity=1000 \ > --wb file=input/monaco.osm.pbf omitmetadata=true This generates a monaco extract from a planet-dump. Also note that I used completeWays=yes and clipIncompleteEntities=true to ensure coastlines are clipped correctly. Troubleshooting FAQ pending ... Category:Browse